In many computer systems, pre-production testing of computer systems, including application-specific computer systems like web servers, is critical to avoiding unexpected and expensive operational problems once the computer systems are deployed and relied upon. General areas of concern include processing bottlenecks and resource use problems (for example, memory and CPU usage). In particular, systems with varying loads (i.e., levels of use) or long-term load carrying may be susceptible to operational problems at higher loads or after operation for long periods of time, although such problems may not be evident in small-scale pre-production testing. A particular, although not exclusive, example of this concept is recognizable in web servers, where the loads on a web server (for example, client-sent file requests) may vary widely according to the number of users accessing the server. Also, extended periods of high server loads may occur (for example, on an e-commerce web server during a peak shopping period).
Conventional computer system testing systems are known for assessing computer system operation. However, conventional testing systems usually perform only a few functions, and do not provide a comprehensive diagnostic package.
For example, load test tools such as LoadRunner® are commercially available from Mercury Interactive of Sunnyvale, Calif. The LoadRunner® test tool is software-implemented and provides a mechanism for running load testing, collecting results, and reporting the collected results. However, it does not select or otherwise help to select specific parametric information to be collected or specific tests to be run. The computer system parameters measured are usually predetermined and not user-selectable. Thus, a system engineer or the like must be able to recognize which test data are of significance.
Similarly, conventional software-implemented application data collection tools are available, for example, from Keynote Systems Inc. However, the Keynote tool is limited to testing/monitoring web sites by deploying agents worldwide that test the client-level experience with respect to a given web site. The Keynote tool only measures user-level response times and throughput, and user-level errors (such as http error codes). It does not monitor a computer system at the server level, nor does it provide any diagnostic functionality. A user must independently decide what tests should be run and must interpret the results.
SE Toolkit is a conventional freeware performance analysis package. SE Toolkit provides system-level diagnosis tools but does not analyze user-experience parameters at an application level. Its primary focus is the computer system—specifically, operating system resource parameters. Analysis of the system parameters is limited to flagging system parameters whose values exceed predetermined thresholds. Some problem-solving functionality is provided, but it is limited to relatively simple problems corresponding to measurement of a single parameter. SE Toolkit provides a language interpreter for creating or editing measurement rules.
Quest Software, Inc. of Irvine, Calif. offers a software package called Spotlight™ that graphically displays server processes and data flow in real time. However, a user must identify congestion and bottleneck problems and decide what action to take. Spotlight™ is application-specific, so a given version of Spotlight™ is limited to analyzing server activity when running a specific application.
Quest also offers a software package called Foglight™ for monitoring application-wise use of system resources. In a particular embodiment, specific software “cartridges” are provided for specific applications to be monitored. Thus, general monitoring of applications with the Quest cartridges is not possible.